The use of remotely controlled robots has seen substantial increase in many fields. For the military and law enforcement, robots are being used to inspect otherwise dangerous or hazardous locations and objects, such as suspected explosive devices or hazardous materials. Moreover, in the area of hazardous environments, such as nuclear facilities, robots are being used to monitor radiation levels and simply to provide users a “set of eyes” within the hazardous environment. Two prominent examples of such robotic systems are dual-arm robots used by the oil and gas industry for remote mobile manipulation on the ocean floor, and the robots used to provide environmental information from within the disabled Fukushima Daiichi nuclear reactor in Japan.
The remote control of such robots is generally undertaken through an image or visual feed, or multiple image or visual feeds, taken from cameras located on the robot and fed back to remote monitors and screens through which the user or robot controller views and interprets the image or visual feeds. Such an image or visual feed and monitor provide limited information to the user. For example the monitor displays a limited field of view image, in that the image provided is only what the camera or video sees. Further, the image presented to the user through the monitor is essentially two-dimensional. Such limitations require the user to expend substantial time and effort to interpret and understand the robot images, and the robot's contextual surroundings.
With respect to robotic control systems, such systems typically use a single fixed reference frame through which the user interprets task requirements and accordingly command inputs. The usual single reference frame is the “world coordinates” or a joint space viewed by the robot. While some research has been undertaken to use “end-effector” coordinates for command interpretation, there has not been a control system infrastructure that permits dynamically alterable control reference frames through which to apply control inputs. Given that the operator has limited situational awareness as noted above, such reference frame limitations result in a substantial mental interpretative load on the operator or user to associated image or visual feedback and appropriate control inputs.
Accordingly, it would be highly desirable to provide methods and systems for enhanced operator situational awareness using improved aggregation of multiple image feeds and intuitive operator control using improved dynamically selectable command reference frames.